At the Australian Antarctic Division we maintain a collection of publically available data sets for general use. To maintain and read these data we use the R for Australian Antarctic Division (RAAD) packages raadsync and raadtools.

This document aims to describe the capabilities of the tools used to build and maintain and use these data sets as well as highlight the exciting new interactive visualizations provided by mapview.

A good example data set is the NSIDC 25km passive microwave sea ice concentration. Here we

The RAAD packages may be installed with devtools from Github (please note these are from different GitHub repositories).

devtools::install_github("AustralianAntarcticDivision/raadtools")
devtools::install_github("AustralianAntarcticDataCentre/raadsync")

Data repository - administrator task

Register a location for the data to be stored locally, this can be anywhere that is writable by the maintainer of the collection.

NOTE Before running this code, please be aware that these tools are designed to download very long time-series of dozens of data sets. They can pull down very many gigabytes of files from only one collection, and for example we have several terabytes because we tend to register all the ones available and have them all completely available and up to date. Not everybody can do this! It’s for shared resources at a large research institute.

That said, the sea ice concentration data is relatively small and can be obtained on its own. The download below is …

Here we put a new file in our local user directory and set it as the default location understood by RAAD. This should be a shared location for general usage, but for this example to be as widely useable as possible a local user installation is reasonable.

dfd <- normalizePath("~/raad", winslash = "/")
dir.create(dfd)
## Warning in dir.create(dfd): 'C:\Users\mdsumner\Documents\raad' already
## exists
options(default.datadir = dfd)

Load raadsync, on the first time you need to confirm a setting for caching if this is being done interactively. Just enter “Y”.

library(raadsync)

Read the built-in default config file and process it for only the sea ice data of interest. We use “NULL” for the local config as we are not overriding any defaults.

cfg <- read_repo_config(local_config_file = NULL)
## Checking config for dataset: NSIDC SMMR-SSM/I Nasateam sea ice concentration ... done.
## Checking config for dataset: NSIDC SMMR-SSM/I Nasateam near-real-time sea ice concentration ... done.
## Checking config for dataset: NSIDC passive microwave supporting files ... done.
## Checking config for dataset: NSIDC SMMR-SSM/I Nasateam sea ice time since melt ... done.
## Checking config for dataset: Nimbus Ice Edge Points from Nimbus Visible Imagery ... done.
## Checking config for dataset: Artist AMSR-E passive microwave sea ice concentration ... done.
## Checking config for dataset: Artist AMSR-E passive microwave supporting files ... done.
## Checking config for dataset: Artist AMSR2 near-real-time passive microwave sea ice concentration ... done.
## Checking config for dataset: Artist AMSR2 regional passive microwave sea ice concentration ... done.
## Checking config for dataset: Artist AMSR2 passive microwave supporting files ... done.
## Checking config for dataset: CERSAT SSM/I sea ice concentration ... done.
## Checking config for dataset: MODIS Composite Based Maps of East Antarctic Fast Ice Coverage ... done.
## Checking config for dataset: Smith and Sandwell bathymetry ... done.
## Checking config for dataset: ETOPO1 ... done.
## Checking config for dataset: ETOPO2 ... done.
## Checking config for dataset: Bedmap2 ... done.
## Checking config for dataset: Kerguelen Plateau Bathymetric Grid 2010 ... done.
## Checking config for dataset: GVDEM2008 ... done.
## Checking config for dataset: Geoscience Australia XYZ multibeam bathymetric grids of the Macquarie Ridge ... done.
## Checking config for dataset: The International Bathymetric Chart of the Southern Ocean (IBCSO) - digital bathymetric model ... done.
## Checking config for dataset: The International Bathymetric Chart of the Southern Ocean (IBCSO) - digital chart for printing ... done.
## Checking config for dataset: RTOPO-1: A consistent dataset for Antarctic ice shelf topography and global ocean bathymetry ... done.
## Checking config for dataset: Radarsat Antarctic Mapping Project Digital Elevation Model Version 2 ... done.
## Checking config for dataset: NOAA Optimum Interpolation 1/4 Degree Daily Sea Surface Temperature Analysis ... done.
## Checking config for dataset: NOAA Optimum Interpolation Sea Surface Temperature V2 ... done.
## Checking config for dataset: NOAA Extended Reconstructed Sea Surface Temperature V3b ... done.
## Checking config for dataset: Oceandata MODIS Terra Level-3 mapped monthly 9km sea surface temperature ... done.
## Checking config for dataset: GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis (v4.1) (GDS versions 1 and 2) ... done.
## Checking config for dataset: Ssalto/Duacs gridded absolute dynamic topography ... done.
## Checking config for dataset: Ssalto/Duacs gridded absolute geostrophic velocities ... done.
## Checking config for dataset: Ssalto/Duacs gridded sea level anomalies ... done.
## Checking config for dataset: Ssalto/Duacs gridded mean and climatological sea level anomalies ... done.
## Checking config for dataset: GSHHG coastline data ... done.
## Checking config for dataset: Polar environmental data layers ... done.
## Checking config for dataset: Oceandata MODIS Aqua Level-3 binned daily remote sensing reflectance ... done.
## Checking config for dataset: Oceandata SeaWiFS Level-3 binned daily remote sensing reflectance ... done.
## Checking config for dataset: NASA MODIS Rapid Response Antarctic Mosaic ... done.
## Checking config for dataset: CSIRO Atlas of Regional Seas 2009 ... done.
## Checking config for dataset: NCEP-DOE Reanalysis 2 6-hourly data ... done.
## Checking config for dataset: NCEP-DOE Reanalysis 2 daily averages ... done.
## Checking config for dataset: NCEP-DOE Reanalysis 2 monthly averages ... done.
## Checking config for dataset: World Ocean Atlas 2009 ... done.

Explore this configuration data set.

Nothing is set to synchronize.

any(cfg$do_sync)
## [1] FALSE

What data sets are about “ice”?

grep("ice", cfg$name, ignore.case = TRUE, value = TRUE)
##  [1] "NSIDC SMMR-SSM/I Nasateam sea ice concentration"                                             
##  [2] "NSIDC SMMR-SSM/I Nasateam near-real-time sea ice concentration"                              
##  [3] "NSIDC SMMR-SSM/I Nasateam sea ice time since melt"                                           
##  [4] "Nimbus Ice Edge Points from Nimbus Visible Imagery"                                          
##  [5] "Artist AMSR-E passive microwave sea ice concentration"                                       
##  [6] "Artist AMSR2 near-real-time passive microwave sea ice concentration"                         
##  [7] "Artist AMSR2 regional passive microwave sea ice concentration"                               
##  [8] "CERSAT SSM/I sea ice concentration"                                                          
##  [9] "MODIS Composite Based Maps of East Antarctic Fast Ice Coverage"                              
## [10] "RTOPO-1: A consistent dataset for Antarctic ice shelf topography and global ocean bathymetry"

We want the NSIDC SMMR-SSMI/I Nasateam sea ice concentration, though we will be selective and only obtain the southern hemisphere and a recent time series to save time and storage. These data are excellent for exploration as they are relatively low-volume data, delivered in a straightforward binary format on native Polar Stereographic map projection, with complete spatial coverage for a complete daily time series from 1979 to now. Obviously the two poles northern and southern are stored separately, we exclude the north by default here.

dnames <- c("NSIDC SMMR-SSM/I Nasateam sea ice concentration", 
            "NSIDC SMMR-SSM/I Nasateam near-real-time sea ice concentration" )
 
myconfig <- subset(cfg, name %in% dnames)
myconfig$local_file_root  <- file.path(dfd, "data")
myconfig$do_sync <- TRUE

Investigate the download options and modify to suit, we only want recent data and the southern hemisphere.

myconfig$method_flags
## [1] "--exclude-directories=pub/DATASETS/nsidc0051_gsfc_nasateam_seaice/final-gsfc/browse,pub/DATASETS/nsidc0051_gsfc_nasateam_seaice/final-gsfc/north --recursive --level=inf"
## [2] "--exclude-directories=pub/DATASETS/nsidc0081_nrt_nasateam_seaice/browse,pub/DATASETS/nsidc0081_nrt_nasateam_seaice/north --recursive --level=inf"
## only this year and last year (probably on 2015 is available for "final" anyway)
myconfig$method_flags[1] <- 
paste(myconfig$method_flags[1], "--accept=\"*nt_2016*\"", "--accept=\"*nt_2015*\"")

## only this year for near-real-time
myconfig$method_flags[2] <- 
paste(myconfig$method_flags[2], "--accept=\"*nt_2016*\"")

Synchronize away, please note that this process is time consuming as it thoroughly checks the remote and local sources, including hash signatures for changed files when possible.

sync_repo(myconfig)

Build the file list cache, this is a convenience mechanism for read functions to save scanning the file system. Administrators, please note that the synchronization and file list caching may be set up as routine system jobs keep everything up to date.

dir.create(file.path(dfd, "admin"))
dir.create(file.path(dfd, "admin/filelist"))
fs1 <- list.files(file.path(getOption('default.datadir'), 'data'), all = TRUE, recursive = TRUE, full.names = TRUE, no.. = TRUE)
fs1 <- normalizePath(fs1, winslash = "/")

fs <- gsub(paste0(getOption('default.datadir'), "/"), "", fs1)
save(fs, file = file.path(dfd, 'admin', 'filelist', 'allfiles2.Rdata'))
writeLines(fs, file.path(dfd, 'admin', 'filelist', 'allfiles2.txt'))

Now check what we have in terms of files.

library(raadtools)
## Loading required package: raster
## Loading required package: sp
icf <- icefiles()
range(icf$date)
## [1] "2016-01-01 GMT" "2016-06-09 GMT"
## size of collection is pretty small given our set limits above
sum(file.info(icf$fullname)$size)/1e6
## [1] 16.93913

Read the data and plot!

ice <- readice(icf$date[seq(1, nrow(icf), by = 14)])
library(mapview)
## Loading required package: leaflet
cubeView(ice)